Gaussian Mixture Model with Local Consistency

Gaussian Mixture Model (GMM) is one of the most popular data clustering methods which can be viewed as a linear combination of different Gaussian components. In GMM, each cluster obeys Gaussian distribution and the task of clustering is to group observations into different components through estimating each cluster’s own parameters. The ExpectationMaximization algorithm is always involved in such estimation problem. However, many previous studies have shown naturally occurring data may reside on or close to an underlying submanifold. In this paper, we consider the case where the probability distribution is supported on a submanifold of the ambient space. We take into account the smoothness of the conditional probability distribution along the geodesics of data manifold. That is, if two observations are “close” in intrinsic geometry, their distributions over different Gaussian components are similar. Simply speaking, we introduce a novel method based on manifold structure for data clustering, called Locally Consistent Gaussian Mixture Model (LCGMM). Specifically, we construct a nearest neighbor graph and adopt Kullback-Leibler Divergence as the “distance” measurement to regularize the objective function of GMM. Experiments on several data sets demonstrate the effectiveness of such regularization. Introduction Clustering is an unsupervised classification of patterns (observations, data items, or feature vectors) into groups(clusters) (Jain, Murty, and Flynn 1999). The goal of it is to organize objects into groups such that members within each group are similar in some way. Therefore, a cluster is a collection of objects which are “close” between them and are “dissimilar” to others belonging to different clusters. Data clustering is one of the common techniques in exploratory data analysis. It has been addressed in many contexts and has drawn enormous attention in many fields, including data mining, machine learning, pattern recognition and information retrieval. The clustering algorithms can be roughly divided into two categories: similarity-based and model-based. Similaritybased clustering algorithms are designed on the basis of similarity function between data observations without any Copyright c © 2010, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. probability assumption. K-means (Duda, Hart, and Stork 2000) and spectral clustering (Ng, Jordan, and Weiss 2001; Shi and Malik 1997) are two representative examples. The former is designed to minimize the sum of distances between the assumed cluster centers and data samples, while the latter usually clusters the data points using the top eigenvectors of graph Laplacian (Chung 1997), which is defined on the affinity matrix of data points. From the graph partitioning perspective, spectral clustering tries to find the best cut of the graph, aiming at optimizing the predefined criterion function. Normalized cut (Shi and Malik 1997) is one of the most well applied criterion functions. Unlike similarity-based methods, model-based clustering can generate soft partition which is sometimes more flexible. Model-based methods use mixture distributions to fit the data and the conditional probabilities are naturally used to assign probabilistic labels. One of the most widely used mixture models for clustering is Gaussian Mixture Model (Bishop 2006). Each Gaussian density is called a component of the mixture and has its own mean and covariance. In many applications, their parameters are determined by maximum likelihood, typically using the Expectation-Maximization algorithm (Dempster, Laird, and Rubin 1977). GMM assumes that the probability distribution generating the data is supported on the Euclidean space. However, many previous studies (Tenenbaum, de Silva, and Langford 2000; Roweis and Saul 2000; Belkin and Niyogi 2001) have shown naturally occurring data may reside on or close to an underlying submanifold. It has also been shown that learning performance can be significantly enhanced if the manifold (geometrical) structure is exploited (Ng, Jordan, and Weiss 2001; Belkin, Niyogi, and Sindhwani 2006; Cai, Wang, and He 2009; Cai, He, and Han 2010). In this paper, we propose a novel model-based algorithm for data clustering, called Locally Consistent Gaussian Mixture Model (LCGMM), which explicitly considers the manifold structure. Following the intuition that naturally occurring data may reside on or close to a submanifold of the ambient space, we incorporate a regularizer into the objective function of Gaussian Mixture Model after constructing a nearest neighbor graph and adopting Kullback-Leibler Divergence as the “distance” measurement. It is important to note that the work presented here is fundamentally based on 512 Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence (AAAI-10)